Preview of Award 2228092 - annual Project Report

Accomplishments

* What are the major goals of the project?

* What was accomplished under these goals and objectives (you must provide information for at least one of the 4 categories below)?

Major Activities:

1) [Activity 1 - Testbed Development:] The project began supporting a new PhD student (Eriana Sobral) in August 2023 to assist with the development of the multi-robot testbed.  The robots and networking infrastructure were purchased, delivered, and initially setup in the PI's lab space.  Ms. Sobral left the project in December 2023 due to health issues, so work on the testbed was temporarily paused during Spring 2024.  Work will recommence in summer 2024.  The project has a new student (Roghayeh Rafiei Sangari) coming in August 2024 to begin supporting this part of the project.  The outcomes from this activity will support GOAL 4 on testbed development.

2) [ Activity 2 - Learning-enabled control of dynamical systems:] This activity sought to complete the learning-enabled model-following control methods discussed in the original project proposal.  That approach was investigated by a Notre Dame B.S. student (Nolan Fey) as part of his BS thesis project.  Mr. Fey graduated from Notre Dame in May 2023 and is now a PhD student at MIT.  After going to MIT, Mr. Fey completed some of the open tasks in his BS thesis where he used the learning-enabled model-following methods on mini-cheetah quadruped robot.  This hardware demonstration was done with the help of his former collaborators at Notre Dame's robotics lab (Li, Adrian, Wensing) using the methods developed by the project PI (Lemmon).  Those results (see product [1]) will be reported in the 6th annual learning for dynamics & control conference (L4DC), July 15-17, 2024, University of Oxford,  England.  This activity supports GOAL 3, in demonstrating the online adaptation of a RL action policy to ensure good performance of the physical (robotic) system.

3) [Activity 3 - Fair Federated Learning:] The original results reported in the project proposal focused primarily on robotic systems (see Activity 2), but this project will extend those methods to networked cyber-physical systems.  This activity, therefore, began looking at how federated learning frameworks might be used for distributed learning of controllers.   One of the PI's PhD students (Yuying Duan) began supporting this activity by investigating post-processing approaches for federated learning.  That work developed methods for minimizing the performance lost when the federated learning framework began enforcing "fair" allocation resource constraints on the system's clients.  These results (see product [2]) were reported in a paper that was submitted to the 2024 International Conference on Machine Learning (ICML), July 21-27, 2024, Vienna Austria.  While this paper was not accepted for presentation at the conference, it is being developed into a more complete arXiv version that will be resubmitted to NeurIPS.    Starting in summer 2024, the PhD student (Yuying Duan) will begin examining how decentralized MARL action policies can be trained in a federated learning framework, with the specific objective of balancing action policy optimality against client fairness.  The outcomes from this activity directly support GOAL 2 (MARL for Digital Twins).  

4) [Activity 4 - Safe Sim2Real Transfer:]. In Spring 2024, the PI and his student (Yuying Duan) began formally developing a method for the safe transfer of reinforcement learning (RL) action policies obtained in a virtual training environment to an unknown physical testing enviornment.   We refer to this transfer between a training and testing environment as Sim2Real transfer.  The main outcome from this activity was the formulation of a provably correct framework for Sim2Real transfer based on a finite abstraction developed from the virtual training environment.  In particular, we found an algorithmically efficient method for learning an abstraction for the training environment that could then be used to generate reference trajectories that would enforce "safe" operation in the physical test environment using a auxiliary controller that could be learned on-line.   The approach is a combination of control barrier function (CBF) [Ames et al, IEEE-TAC, 62(8):3861-3876, 2016] and predictive safety filter (PSF) [Wabersich et al. Automatica, 129:109597, 2021] where the CBF is learned from the training environment's finite abstraction and the PSF is generated using the DMDc methods [Proctor et al., SIAM Journal of Applied Dynamical Systems, 15(1): 142-161, 2016] we used in [Lemmon et al, American Control Conference, 2022] and [Fey et al. L4DC conferencd, 2024].  This expanded framework for Sim2Real Transfer is currently being finalized and initial reports on the outcomes will be prepared over Summer 2024.  A new PhD student (Roghayeh Rafiei Sangari) will begin assisting the PI on this activity in August 2024.   This activity supports GOAL 3 of the project.  

Specific Objectives:

Specific Objectives:

[Objective 1:] Develop hardware multi-robot testbed (GOAL 4)
Status: hardware purchased and initial networking of robots complete.  Hiring new student assistants to help with testbed development

[Objective 2:] Develop Meta-learning framework for identifying challenging scenarios used in MARL training of digital twins (GOAL 1):  
status: Initial project proposed using meta-learning for robust control where the parameters were the weights on the effort and tracking error of the control objective.  Initial empirical studies did not demonstrate that this was a promising use of meta-learning.  It became apparent that the game-theoretic nature of meta-learning was much more approrpiate for identifying adversarial scenarios for MARL, so future work will look in this direction.

[Objective 3:] Integration of decentralized MARL with federated learning framework (GOAL 2)

status: initial work on federated learning suggests that post-processing of action policies is an algorithmically efficient way of optimizing digital twin performance subject to fairness constraints.  Future work will look explicitly at how a mean-field MARL action policies can be trained in the federated learning framework.  

[Objective 4:] Develop an online method for Sim2Real Transfer (GOAL 3):

status: The formal basis for safe Sim2Real transfer were established in Spring 2024 semester.  Future work will experimentally study the performance of the approach using the multi-robot testbed as a final proof of concept.  

Significant Results:

Nothing to report

Key outcomes or Other achievements:

Nothing to report

* What opportunities for training and professional development has the project provided?

* Have the results been disseminated to communities of interest? If so, please provide details.

* What do you plan to do during the next reporting period to accomplish the goals?

Back to the top